Pose estimation method and apparatus

ABSTRACT

A three-dimensional image data is formulated and saved in a memory for indicating a three-dimensional shape of an object and reflectivity or color at every point of the object. For each of multiple pose candidates, an image space is created for representing brightness values of a set of two-dimensional images of the object which is placed in the same position and orientation as the each pose candidate. The brightness values are those which would be obtained if the object is illuminated under varying lighting conditions. For each pose candidate, an image candidate is detected within the image space using the 3D model data and a distance from the image candidate to an input image is determined. Corresponding to the image candidate whose distance is smallest, one of the pose candidates is selected. The image space is preferably created from each of a set of pose variants of each pose candidate.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The present invention relates generally to the estimation of theposition and orientation of a three-dimensional object in atwo-dimensional image taken by an imaging device such as a camera, andmore specifically to a pose estimation method and apparatus forestimating the pose of an object using the three-dimensional shape of atarget object and its surface reflectivity or color information.

[0003] 2. Description of the Related Art

[0004] Various methods are known in the art for estimating the positionand orientation of a three-dimensional object in a two-dimensionalimage. One approach is the analytic solution of the n-point perspectiveproblem which concerns the determination of the position and orientation(pose) of a camera with respect to a three-dimensional object when npoints of the object corresponds to n points of a two-dimensional image.The issue of computations associated with pose estimation is describedin documents such as “An Analytic Solution for the Perspective 4-pointProblem”, Radu Horaud et al., Computer Vision, Graphics and ImageProcessing, 47, pp. 33-44 (1989) and “Linear N≧4-Point PoseDetermination”, Long Quan and Zhongdan Lan, Proceedings of IEEEInternational Conference Computer Vision, 6, pp. 778-783 (1998). Theanalytic solution of the n-point perspective problem is the problem offinding the best pose from a plurality of candidates that werecalculated from feature points of an image object in an input image andthe registered positions of corresponding feature points of athree-dimensional target object. If correspondence occurs at least threefeature points, a maximum of fourth pose candidates can be calculated.Since the amount of positional information contained in the three-pointfeatures is not sufficient to uniquely determine the best candidate,correspondences are usually required for at least four feature points.Pose candidates are first calculated from three of the at least fourpoints and a best one is selected from the candidates when the remainderpoint is calculated. However, in so far as an error is contained in thepositional information there are no pose parameters where correspondenceoccurs at all feature points. Since this type of error is unavoidableand no information other than the position data is used, the analyticsolution of n-point perspective problem can be considered as an errorminimization technique such as the least squares algorithm. If thetarget object has jaggy edges or an ambiguous shape or has no particularsurface features, the error increases and detectable features decrease,

[0005] Japanese Patent Publication 2001-283229 discloses anothertechnique in which errors are designed into the feature points positiondata and pose determination involves the use of only those featurepoints where errors are small. According to this technique, a set ofarbitrarily combined three feature points is selected and a posecandidate calculation is performed to obtain pose candidates, whilecorrecting the position of each selected point. The pose candidates arethen fitted to all feature points and the best candidate is chosen thatyields a minimum error.

[0006] Japanese Patent Publication 2000-339468 discloses a technique inwhich an object is subjected to illumination at different angles. A pairof 3D shapes of the same object as seen from a given viewing angle isproduced and these 3D shapes are searched for corresponding featurepoints. The detected feature points are used to estimate the differencebetween the two 3D shapes. This technique can be used for poseestimation by producing an input image from one of the 3D shapes.

[0007] Another prior art technique disclosed in Japanese PatentPublication 1999-051611 relates to the detection of contour lines of animage object such as a cylinder. The detected contour lines are comparedwith stored contour line data of a 3D model to correct calculated poseparameters.

SUMMARY OF THE INVENTION

[0008] It is therefore an object of the present invention to provide apose estimation method and apparatus capable of precisely estimating thepose of an object of an image even though the image object was subjectedto variations in shape, surface textures, position and orientation andillumination.

[0009] Another object of the present invention is to provide a poseestimation method and apparatus capable of uniquely estimating the poseof an image object having only three feature points or four featurepoints which are located in exceptional 3D positions.

[0010] According to a first aspect of the present invention, there isprovided a pose estimation method using a plurality of pose candidates.The method comprises the steps of (a) formulating 3D model dataindicating a three-dimensional shape of an object and reflectivity orcolor at every point of the object, (b) creating, for each of the posecandidates, an image space representing brightness values of a pluralityof two-dimensional images of the object which is placed in the sameposition and orientation as the each pose candidate, wherein thebrightness values would be obtained if the object is illuminated undervarying lighting conditions, (c) detecting, for each of the posecandidates, an image candidate within the image space by using the 3Dmodel data and determining a distance from the image candidate to aninput image, and (d) selecting one of the pose candidates whichcorresponds to the image candidate whose distance to an input image issmallest. Preferably, a plurality of pose variants are created from eachof the pose candidates such that each pose variant is displaced inposition and orientation by a predetermined amount from the posecandidate. Using each of the pose variants, the step (c) creates theimage space.

[0011] According to a second aspect, the present invention provides apose estimation method using a plurality of pose candidates stored in amemory. The method comprises the steps of (a) formulating 3D model dataindicating a three-dimensional shape of an object and reflectivity orcolor at every point of the object, (b) successively reading one of thepose candidates from the memory, (c) creating a plurality of posevariants from the one pose candidate such that each of the pose variantsis displaced in position and orientation by a predetermined amount fromthe one pose candidate, (d) creating, for each of the pose variants, animage space representing brightness values of a plurality oftwo-dimensional images of the object placed in the same position andorientation as the each pose variant, wherein the brightness valueswould be obtained if the object is illuminated under varying lightingconditions, (e) detecting, for each the pose variant, an image candidatewith the image space by using the 3D model data and determining adistance from the image candidate to an input image, (f) repeating steps(b) to (e) to produce a plurality of the image candidates for each ofthe pose candidates, (g) selecting one of the pose candidatescorresponding to the image candidate whose distance to the input imageis smallest, (h) comparing the pose candidate selected by step (g) witha previously selected pose candidate, and (i) replacing the previouslyselected pose candidate with the pose candidate currently selected bystep (g) if the currently selected pose candidate is better than thepreviously selected pose candidate, and repeating steps (b) to (g) untilthe previously selected pose candidate is better an the currentlyselected pose candidate,

[0012] According to a third aspect, the present invention provides apose estimation method comprising the steps of (a) formulating 3D modeldata indicating a three-dimensional shape of an object and reflectivityor color at every point of the object, (b) extracting feature pointsfrom the object and extracting feature points from an input image,. (c)creating a plurality of pose candidates from the extracted featurepoints of the object and the extracted feature points of the input imageand storing the pose candidates in a memory, (d) creating, for each ofthe pose candidates, an image space representing brightness values of aplurality of two-dimensional images of an object placed in the sameposition and orientation as the each pose candidate, wherein thebrightness values would be obtained if the image object is illuminatedunder varying lighting conditions, (e) detecting, for each of the posecandidates, an image candidate within the image space by using the 3Dmodel data and determining a distance from the image candidate to theinput image, and (f) selecting one of the pose candidates correspondingto the image candidate whose distance to the input image is smallest.Preferably, a plurality of pose variants are created from each of thepose candidates such that each pose variant is displaced in position andorientation by a predetermined amount from the pose candidate. Usingeach of the pose variants, the step (d) creates the image space.

[0013] According to a fourth aspect, the present invention provides apose estimation method comprising the steps of (a) formulating 3D modeldata indicating a three-dimensional shape of an object and reflectivityor color at every point of the object, (b) extracting feature pointsfrom the object and extracting feature points from an input image, (c)estimating a possible error of the extracted feature points of the inputimage, (d) creating a plurality of pose candidates from the extractedfeature points of the object and the extracted feature points of theinput image and storing the pose candidates in a memory, (e)successively reading one of the pose candidates from the memory, (f)creating a plurality of pose variants from the one pose candidate suchthat each of the pose variants is displaced in position and orientationby a predetermined amount from the one pose candidate over a rangedetermined by the possible error estimated by step (c), (g) creating,for each of the pose variants, an image space representing brightnessvalues of a plurality of two-dimensional images of the object placed inthe same position and orientation as the each pose variant, wherein thebrightness values would be obtained if the image object is illuminatedunder varying lighting conditions, (h) detecting, for each the posevariant, an image candidate within the image space by using the 3D modeldata and determining a distance from the image candidate to the inputimage, (i) repeating steps (e) to (h) to produce a plurality of theimage candidates for each of the pose candidates, and (j) selecting oneof the pose candidates corresponding to the image candidate whosedistance to the input image is smallest.

BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present invention will be described in detail further withreference to the following drawings, in which:

[0015]FIG. 1 is a block diagram of a pose estimation system according toa first embodiment of the present invention;

[0016]FIG. 2 is a flowchart of the operation of a processor during a 3Dmodel registration routine of the first embodiment of the presentinvention;

[0017]FIG. 3 is a flowchart of the operation of the processor during apose estimation routine according to the first embodiment of the presentinvention;

[0018]FIG. 4 is a flowchart of the operation of the processor during apose estimation routine according to a modified form of the firstembodiment of the present invention;

[0019]FIG. 5 is a flowchart of the operation of the processor during apose estimation routine according to a further modification of the firstembodiment of the present invention;

[0020]FIG. 6 is a block diagram of a pose estimation system according toa second embodiment of the present invention;

[0021]FIG. 7 is a flowchart of the operation of the processor during a3D model registration routine of the second embodiment of the presentinvention;

[0022]FIG. 8 is a flowchart of the operation of the processor during apose estimation routine according to the second embodiment of thepresent invention;

[0023]FIG. 9 is a flowchart of the operation of the processor during apose estimation routine according to a modification of the secondembodiment of the present invention;

[0024]FIG. 10 is an illustration of a 3D-to-2D conversion process forprojecting the coordinates of a 3D object to a 2D plane; and

[0025]FIG. 11 is an illustration of facial feature points.

DETAILED DESCRIPTION

[0026] Referring now to FIG. 1, there is shown a pose estimation systemaccording to a first embodiment of the present invention. The system iscomprised of a 3D scanner 11 for scanning a three-dimensional referenceobject, or 3D model 10 and measuring the 3D model (shape) of the targetobject and the reflectivity of its surface. A target object 12 whoseposition and orientation (pose) is to be estimated is captured by acamera 13 as an image object 19 in a two-dimensional image 14, which isthen fed into a processor 15. Further associated with the processor 15are a number of memories, including a 3D model memory 16, a posecandidates memory 17 in which pose candidates created in a known mannerare stored. A pose variants memory 18 is further connected to theprocessor 15. This memory is used during a pose estimation routine totemporarily store a plurality of pose variants which are created fromeach pose candidate read out of the pose candidate memory.

[0027] During a 3D model registration routine, the processor 15 controlsthe scanner 11 to scan the surface of the reference object 10.

[0028] One example of the 3D scanner 11 is described in Japanese PatentPublication 2001-12925 (published Jan. 19, 2001). According to thisprior art, a light spot with a brightness pattern of sinusoidaldistribution is scanned across the surface of the reference object. Thephase of the sinusoidal distribution is repeatedly varied in incrementsof 2π/4 radian in response to a phase control signal supplied from the3D scanner 11. A pair of cameras are used to detect rays reflecting atdifferent angles off the surface of the reference object 10. Imagesobtained from these cameras are processed to determine a 3D model(shape).

[0029] As described in detail later, the processor 15 receives the 3Dshape data from the scanner 11 as well as texture (or shades of colors)images as a representation of the reflectivity of the surface ofreference object 10,

[0030] During a pose estimation routine, the processor 15 uses the datastored in the memories 16 and 17 to determine a pose candidate that bestfits the image object 19.

[0031] According to the first embodiment, the processor 15 operates in anumber of different modes as will be described below with the aid offlowcharts of FIGS. 2 to 5.

[0032] In FIG. 2, the 3D model registration routine begins with step 201in which the processor 15 receives the output of the 3D scanner 11indicating the 3D model of reference object 10 and determines thereflectivity of the surface of the object 10 from the texture imageadditionally supplied from the 3D scanner 11.

[0033] Processor 15 performs texture creation step 202. Consider asphere 30 (see FIG. 10) whose cubic center coincides with the center ofgravity of a reference object 31. Processor uses the 3D model data todefine texture coordinates (s, t) in terms of latitude and longitude atevery point Q(s, t) on the surface of the sphere 30 by projecting Q(s,t) onto a corresponding point P(x, y, z) on the surface of the referenceobject 31. Then, the processor 15 creates textures T(s, t) thatrepresent the brightness value of every pixel point P(s, t) of atwo-dimensional image 32 by varying the illumination on the referenceobject 31.

[0034] More specifically, the processor 15 uses the Lambertian model toapproximate the reflectivity of the surface of the reference object byassuming that the point source of light is located at an infinitedistance away. Initially, the processor 15 uses the 3D shape data storedin the memory 16 to calculate the normal vector {right arrow over(n)}(s,t) at every point P(s,t). If the reflectivity at point P(s,t) isdenoted as d(s,t) and the light from the point source is represented bya vector {overscore (L)}_(k) (where (k=1, 2, . . . , N), the brightnessT_(k)(s, t) at point P is given by Equation (1):

T _(k)(s,t)=d(s,t)e _(k)(s,t){right arrow over (n)}(s,t){right arrowover (L)} _(k)  (1)

[0035] where, e_(k)(s, t) is 0 if point P(s, t) is shadowed and 1otherwise. This is accomplished by making a decision as to whether ornot a line extending from the point source of light to the point Pcrosses the object. A technique known as ray tracing can be used to makethis shadow-illumination decision. By varying the location of the pointsource of light, a number of brightness values {T_(k)(s, t)} areobtained for a number of different values of index “k”.

[0036] Processor 15 performs basis texture calculation subroutine 210 toapproximate the whole textures with basis textures. In this subroutine,the brightness values are used to calculate a set of basis textures{G_(i)} which represents the basis of a subspace that encompasses mostof the whole set of textures under a given set of illumination vectors{right arrow over (L)}₁, . . . , {overscore (L)}_(N). Processor 15performs this subroutine by first calculating, at step 203, thecovariance matrix of the brightness values of the textures to obtaineigenvectors {overscore (g)}_(i) and eigenvalues λ_(i) of the covariancematrix. The eigenvalues are arranged in a descending order i=1, 2, . . ., N. Processor 15 then extracts “n” eigenvectors from the calculatedeigenvalues {{right arrow over (g)}_(i)} as basis textures {G_(i)},where i=1, 2, . . . , n. The integer “n” is determined by solvingEquation (2). $\begin{matrix}{{\sum\limits_{i = 1}^{n}\quad \lambda_{i}} \geq {R{\sum\limits_{i = 1}^{N}\quad \lambda_{i}}}} & (2)\end{matrix}$

[0037] where, R is a cumulative contributing factor R of eigenvalues. Avalue 0.99 (=99%), for example, is used for the contributing factor R.

[0038] In this way, the processor 15 creates, during the 3D modelregistration routine, 3D model data representing a three-dimensionalshape of the reference object 10 and the basis textures {G_(i)}.

[0039] The following is a description of a number of pose estimationroutines according to the first embodiment of the present invention withreference to the flowcharts of FIGS. 3, 4 and 5.

[0040] The pose estimation routine of the first embodiment is shown inFIG. 3. At step 301, the processor 15 reads an input image from thecamera 15 and reads 3D model data and basis texture data from the 3Dmodel memory 16. At step 302, the processor reads a pose candidate P_(j)from the pose candidates memory 17. Then, the processor 15 performsillumination variation space creation subroutine 320 and imagecomparison subroutine 330 in succession.

[0041] In the illumination variation space creation subroutine 320, theprocessor creates, for each of the pose candidates, an image spacerepresenting the brightness values of a plurality of two-dimensionalimages of the object, which is placed in the same position andorientation as each of the pose candidates and illuminated under varyinglighting conditions.

[0042] More specifically, at step 303 of the illumination variationspace creation subroutine 320, the processor 15 performs a conversionprocess f; (u, v)→(s, t) from the input image to the 3D model so thatevery pixel (u, v) of the input image has its corresponding position (s,t) on the surface of the 3D model. Obviously, this conversion isperformed only on the area of the image where the object is present. Oneway of doing this is to employ a conventional computer graphic techniqueby setting the color of each pixel on the reference object so that itindicates the texture coordinates (s, t) of that point, whereby thecorresponding relationship (u, v)→(s, t) can be obtained from the colorof each pixel of the computer-graphic image. In the conventionalpersonal computer, for example, the RGB components of a full-colorpicture can be set in the range between 0 and 255, the texturecoordinate system (s, t) is in the range between −90 and +90 andtherefore the following color values are set into the texturecoordinates (s, t):

R=(s+90)/180*255  (3a)

G=(t+90)/180*255  (3b)

[0043] From the color of each pixel point (u, v) of an image thusobtained, the texture coordinate system (s, t) can be calculated byusing Equations (3a) and (3b).

[0044] At step 305, the processor 15 uses the above-mentioned coordinateconversion to calculate Equation (4) so that the basis textures areprojected onto the image space “S” of the pose candidate and a set ofbasis images {Bi} is produced as the set of bases of illuminationvariation space “S”:

B _(i)(u, v)=G _(i)(f(u, v)), where i=1, 2, . . . , n  (4)

[0045] At step 306 of the image comparison subroutine 330, the processorcreates an image candidate in which an object of the same orientation asthe pose candidate is present. This is accomplished by using the leastsquares algorithm to calculate the following Equation (5) to identify animage candidate {overscore (C)}_(j) within the space “S” that is nearestto the input image {right arrow over (I)}: $\begin{matrix}{{\overset{\_}{C}}_{j} = {\sum\limits_{i = 1}^{n}\quad {a_{i}{\overset{\_}{b}}_{i}}}} & (5)\end{matrix}$

[0046] where {a_(i)}=arg(min|{right arrow over (I)}-{right arrow over(C)}_(j)|) and B_(i)=basis images. At step 307, a differenceD_(j)=|{right arrow over (I)}-{right arrow over (C)}_(j)| is determinedfor the current pose candidate P_(j). Therefore, step 307 determines thedistance from the image candidate C_(j) to the input image.

[0047] Steps 302 to 307 are repeated until the address pointer “j” isincremented to its maximum value (steps 308, 309). When the maximumvalue of address pointer “j” is reached, flow proceeds to step 310 toselect the image candidate whose distance D_(j) to the input image issmallest.

[0048] A modified form of the first embodiment is shown in FIG. 4, inwhich steps corresponding in significance to those in FIG. 3 are markedwith the same numerals and the description thereof is omitted forsimplicity.

[0049] In FIG. 4, step 302 is followed by step 401 in which theprocessor 15 creates a plurality of variants V_(k) of the pose candidateP_(j) which are respectively given axial displacements of specifiedvalue within prescribed X, Y, Z ranges, and further given angulardisplacements of specified value within prescribed X, Y, Z angularranges. The amount of each displacement is determined by taking intoaccount possible errors associated with each pose candidate when it wasoriginally calculated.

[0050] For example, assume that the pose candidate P_(j) is located atTx=0 mm, Ty=50 mm and Tz=100 mm in the three-dimensional X, Y and Z axesand oriented at Rx=0°, Ry=20°, Rz=40° in the three-dimensional X, Y, Zangles. Further, assume that the possible error is 5 mm or less in theaxial directions and 5 degrees or less in the angular orientations, andthat each of the prescribed ranges of axial displacements is 20 mm andeach of the prescribed ranges of angular displacements is 10 degrees.

[0051] The pose candidate P_(j) is given axial displacements inincrements of ΔTx=5 mm over the range between −10 mm and +10 mm, inincrements of ΔTy=5 mm over the range between 40 mm and 60 mm, and inincrements of ΔTz=5 mm over the range between 90 mm and 110 mm, andfurther given angular displacements in increments of ΔRx=5° over therange between −5° and +5°, in increments of ΔRy=5° over the rangebetween 15° and 25° and in increments of ΔRz=5° over the range between35° and 45°. As a result of the axial and angular displacements, a totalof 5³×3³=3375 pose variants V_(k) (where k=3375) are created. These posevariants are stored in the pose variants memory 18.

[0052] Each of the pose variants is retrieved from the memory 18 at step402 and used in the subsequent illumination variation space creationsubroutine 320 and image comparison subroutine 330 which is followed bydecision step 403.

[0053] At decision step 403, the processor checks to see if all posevariants have been retrieved and tested. If not, flow proceeds to step404 to increment the address pointer of the memory 18 by one and returnsto step 402 to read the next variant to repeat the testing process untilall pose variants are tested. When all pose variants have been tested,the decision at step 403 is affirmative and flow proceeds to decisionstep 308 to determine whether to return to step 302 to repeat theprocess again on the next pose candidate P_(j) or proceed to imagecandidate selection step 310.

[0054] It is seen that, for each pose candidate, comparison tests aremade at close intervals centered about the original position of the posecandidate. In this way, an image candidate can be precisely determined.

[0055] A modification of the pose estimation of FIG. 4 is shown in FIG.5, in which steps corresponding in significance to those in FIGS. 3 and4 are marked with the same numerals. This modification is a simple andyet effective approach by decreasing the amount of computations. In tsmodification, step 302 is followed by variants creation step 501 thatreplaces step 401 of FIG. 4.

[0056] In this variants creation step 501, the processor 15 creates aplurality of variants V_(k) of the pose candidate P_(j) which arerespectively given axial displacements of specified value, and furthergiven angular displacements of specified value.

[0057] For example, assume that the pose candidate P_(j) is located atTx=0 mm, Ty=50 mm and Tz=100 mm in the three-dimensional X, Y and Z axesand oriented at Rx=0°, Ry=20°, Rz=40° in three-dimensional X, Y, Zangles. In this modification, each of the axial displacements ΔTx, ΔTyand ΔTz is set equal to 1 mm and each of the angular displacements ΔRx,ΔRy and ΔRz is set equal to 1 degree.

[0058] The pose candidate P_(j) is given axial displacements ofΔTx=ΔTy=ΔTz=±1 mm, and further given positive and negative angulardisplacements ΔRx=ΔRy=ΔRz=±1°. As a result, a total of 2×6=12 posevariants V_(k) (where k=12) are created. These pose variants are storedin the pose variants memory 18.

[0059] Variant retrieval step 402 subsequently reads a pose variantV_(k) from the pose variants memory 18, which is processed throughsubroutines 320 and 330 until all variants are tested (step 403).

[0060] When all pose variants have been tested, pose candidate selectionstep 310 is performed to select an image candidate whose distance D_(j)to the input image is smallest. Plow proceeds to step 502 which comparesthe currently selected pose candidate with a previously selected posecandidate. If the current pose candidate is better than the previousone, the decision is affirmative at step 502 and flow proceeds to step503 to replace the previously selected pose candidate with the currentlyselected pose candidate. The address pointer of the pose candidatememory 18 is then incremented (step 309) to read the next pose candidatefrom the memory 18 to repeat the same process on the next posecandidate;

[0061] If the currently selected pose candidate is not better than thepreviously selected pose candidate, the decision at step 502 isnegative. In this case, the processor 15 determines that the currentlyselected pose candidate is the best candidate and terminates the poseestimation routine.

[0062] A pose estimation system according to a second embodiment of thepresent invention is shown in FIG. 6. In this embodiment, the featurepoints of an input image extracted and stored in a feature points memory20 during a 3D model registration routine, and pose candidates arecreated from the stored feature points data during a pose estimationroutine and stored in a pose candidates memory 21.

[0063] As shown in FIG. 7, the 3D model registration routine of thesecond embodiment differs from the 3D model registration routine of thefirst embodiment by the inclusion of steps 701 and 702.

[0064] Following the execution of step 201, the routine proceeds to step701 to enter 3D feature points of texture as indicated by symbols+inFIG. 11, or feature points of 3D shape data, and the entered featurepoints are saved in the 3D model memory 16 (step 702). Step 702 isfollowed by texture creation subroutine 220, basis texture calculationsubroutine 230, and data saving step 213.

[0065] As shown in FIG. 8, the pose estimation routine of the secondembodiment is a modification of the pose estimation routine of FIG. 3.At step 801, the processor 15 reads an input image from the camera 15,and reads 3D model and basis texture data from the 3D model memory 16and feature points data from the feature points memory 20.

[0066] At step 802, the processor extracts feature points from the inputimage obtained by the camera 13, and proceeds to step 803 to calculate aplurality of pose candidates such that in each of the pose candidatesthe feature points of the 3D model are projected onto the extractedfeature points of the input image in a manner as described in thedocument “An Analytic Solution for the Perspective 4-point Problem”,Radu Horaud et al., Computer Vision, Graphics and Image Processing, 47,pp. 33-44 (1989). The pose candidates created in this way are saved inthe pose candidates memory 21.

[0067] Step 803 is followed by step 302 for reading a pose candidateP_(j) from the pose candidates memory 21. Then, the processor 15performs illumination variation space creation subroutine 320 and imagecomparison subroutine 330 on the pose candidate P_(j) until all posecandidates are tested and selects the best image candidate at step 310when all the candidates have been read and compared.

[0068] The pose estimation routine according to a modification of thesecond embodiment of Fig, 8 is shown in Fig, 9, in which stepscorresponding to those in FIG. 8 are marked with the same numerals asthose used in FIG. 8.

[0069] In FIG. 9, the processor executes error estimation step 901 afterthe feature points extraction step 802 is performed to estimate possibleerror of the extracted feature points. Step 901 is then followed by thepose candidate calculation step 803 to create a plurality of posecandidates from the feature points and saved in the pose candidatesmemory 21, as discussed above. As in FIG. 8, step 302 follows step 803to read a pose candidate P_(j) from the pose candidates memory 21.

[0070] However, this modified embodiment introduces step 902 to createpose variants V_(k) of candidate P_(j) by using the estimated error ofthe candidate P_(j). Specifically, the pose variants are given X, Y, Zaxial displacements of specified value within X, Y, Z axial ranges whichare determined by the estimated error of candidate P_(j) and are furthergiven X, Y, Z angular displacements of specified value within X, Y, Zangular ranges which are also determined by the estimated error ofcandidate P_(j). The pose variants so created are stored in the variantsmemory 18.

[0071] If the facial feature points of FIG. 11, as numbered 1, 4 and 10,have been entered at step 701 (FIG. 7) and the error has been estimatedby step 901 as smaller than five pixels, both axial and angulardisplacements are such as to move the pose candidate randomly within theradius of five pixels, producing as many as 100 variants for each posecandidate, for example.

[0072] At step 903, the processor reads a variant V_(k) from thevariants memory 18 and executes subroutines 320 and 330 on the read posevariant to create an image candidate. The address pointer of the memory18 is successively incremented (step 905) to repeat the process on allthe pose variants (step 904). In his way, a plurality of comparisonresults are produced for a pose candidate read from the pose candidatememory 21. The processor repeatedly executes steps 308 and 309 to repeatthe same process until all pose candidates P are read from the memory 21for outputting the best pose candidate at step 310.

What is claimed is:
 1. A pose estimation method by using a plurality ofpose candidates, comprising the steps of: a) formulating 3D model dataindicating a three-dimensional shape of an object and reflectivity orcolor at every point of said object; b) creating, for each of said posecandidates, an image space representing brightness values of a pluralityof two-dimensional images of said object which is placed in the sameposition and orientation as said each pose candidate, wherein saidbrightness values would be obtained if said object is illuminated undervarying lighting conditions; c) detecting, for each of said posecandidates, an image candidate within said image space by using said 3Dmodel data and determining a distance from the image candidate to aninput image; and d) selecting one of the pose candidates whichcorresponds to the image candidate whose distance to an input image issmallest.
 2. The method of claim 1, wherein step (a) comprises the stepsof: measuring reflectivity or color at every point of said object undervarying illuminations to produce a plurality of illumination variationtextures each representing a brightness value of said every point ofsaid object; calculating basis textures approximating said plurality ofillumination variation textures; and formulating said 3D model data sothat the model data indicates said basis textures as well as saidthree-dimensional shape.
 3. The method of claim 1, wherein step (c)detects said image candidate when distance from the image candidate tosaid input image is nearest within said image space.
 4. The method ofclaim 3, wherein step (c) uses the least squares algorithm to detectsaid image candidate.
 5. The method of claim 1, further comprising thestep of creating a plurality of pose variants from each of said posecandidates such that each of the pose variants is displaced in positionand orientation by a predetermined amount from said each pose candidate,and wherein step (b) creates said image space for each of said posevariants.
 6. The method of claim 5, wherein each of said pose variantsis axially and rotationally displaced by said predetermined amount in athree-dimensional coordinate system over a specified range centeredabout said each pose candidate.
 7. The method of claim 6, wherein saidspecified range is determined based on a possible error of said eachpose candidate.
 8. A pose estimation method by using a plurality of posecandidates stored in a memory, comprising the steps of: a) formulating3D model data indicating a three-dimensional shape of an object andreflectivity or color at every point of said object; b) successivelyreading one of said pose candidates from said memory; c) creating aplurality of pose variants from said one pose candidate such that eachof the pose variants is displaced in position and orientation by apredetermined amount from said one pose candidate; d) creating, for eachof said pose variants, an image space representing brightness values ofa plurality of two-dimensional images of said object placed in the sameposition and orientation as said each pose variant, wherein saidbrightness values would be obtained if said object is illuminated undervarying lighting conditions; e) detecting, for each said pose variant,an image candidate within said image space by using said 3D model dataand determining a distance from the image candidate to an input image;f) repeating steps (b) to (e) to produce a plurality of said imagecandidates for each of said pose candidates; g) selecting one of thepose candidates corresponding to the image candidate whose distance tosaid input image is smallest; h) comparing the pose candidate selectedby step (g) with a previously selected pose candidate; and i) replacingthe previously selected pose candidate with the pose candidate currentlyselected by step (g) if the currently selected pose candidate is betterthan the previously selected pose candidate, and repeating steps (b) to(g) until the previously selected pose candidate is better than thecurrently selected pose candidate.
 9. The method of claim 8, whereinstep (a) comprises the steps of: measuring reflectivity or color atevery point of said object under varying illuminations to produce aplurality of illumination variation textures each representing abrightness value of said every point of said object; calculating basistextures approximating said plurality of illumination variationtextures; and formulating said 3D model data so that the image dataindicates said basis textures as well as said three-dimensional shape.10. The method of claim 8, wherein step (e) detects said image candidatewhen distance from the image candidate to said input image is nearestwithin said image space.
 11. The method of claim 8, wherein step (e)uses the least squares algorithm to detect said image candidate.
 12. Apose estimation method comprising the steps of: a) formulating 3D modeldata indicating a three-dimensional shape of an object and reflectivityor color at every point of said object; b) extracting feature pointsfrom said object and extracting feature points from an input image; c)creating a plurality of pose candidates from the extracted featurepoints of said object and the extracted feature points of said inputimage and storing the pose candidates in a memory; d) creating, for eachof said pose candidates, an image space representing brightness valuesof a plurality of two-dimensional images of an object placed in the sameposition and orientation as said each pose candidate, wherein saidbrightness values would be obtained if said image object is illuminatedunder varying lighting conditions; e) detecting, for each of said posecandidates, an image candidate within said image space by using said 3Dmodel data and determining a distance from the image candidate to saidinput image; and f) selecting one of the pose candidates correspondingto the image candidate whose distance to said input image is smallest.13. The method of claim 12, wherein step (a) comprises the steps of:measuring reflectivity or color at every point of said object undervarying illuminations to produce a plurality of illumination variationtextures each representing a brightness value of said every point ofsaid object; calculating basis textures approximating said plurality ofillumination variation textures; and formulating said 3D model data sothat the image data indicates said basis textures as well as saidthree-dimensional shape.
 14. The method of claim 12, wherein step (e)detects said image candidate when distance from the image candidate tosaid input image is nearest within said image space.
 15. The method ofclaim 12, wherein step (e) uses the least squares algorithm to detectsaid image candidate.
 16. The method of claim 12, further comprising thestep of creating a plurality of pose variants from each of said posecandidates such that each of the pose variants is displaced in positionand orientation by a predetermined amount from said each pose candidate,and wherein step (c) creates said image space for each of said posevariants.
 17. The method of claim 16, wherein each of said pose variantsis axially and rotationally displaced by said predetermined amount in athree-dimensional coordinate system over a specified range centeredabout said each pose candidate.
 18. The method of claim 17, wherein saidspecified range is determined based on a possible error of said eachpose candidate.
 19. A pose estimation method comprising the steps of: a)formulating 3D model data indicating a three-dimensional shape of anobject and reflectivity or color at every point of said object; b)extracting feature points from said object and extracting feature pointsfrom an input image; c) estimating a possible error of the extractedfeature points of said input image; d) creating a plurality of posecandidates from the extracted feature points of said object and theextracted feature points of said input image and storing the posecandidates in a memory; e) successively reading one of said posecandidates from said memory; f) creating a plurality of pose variantsfrom said one pose candidate such that each of the pose variants isdisplaced in position and orientation by a predetermined amount fromsaid one pose candidate over a range determined by said possible errorestimated by step (c); g) creating, for each of said pose variants, animage space representing brightness values of a plurality oftwo-dimensional images of said object placed in the same position andorientation as said each pose variant, wherein said brightness valueswould be obtained if said image object is illuminated under varyinglighting conditions; h) detecting, for each said pose variant an imagecandidate within said image space by using said 3D model data anddetermining a distance from the image candidate to said input image; i)repeating steps (e) to (h) to produce a plurality of said imagecandidates for each of said pose candidates; and j) selecting one of thepose candidates corresponding to the image candidate whose distance tosaid input image is smallest.
 20. The method of claim 19, wherein step(a) comprises the steps of: measuring reflectivity or color at everypoint of said object under varying illuminations to produce a pluralityof illumination variation textures each representing a brightness valueof said every point of said object; calculating basis texturesapproximating said plurality of illumination variation textures; andformulating said 3D model data so that the image data indicates saidbasis textures as well as said three-dimensional shape.
 21. The methodof claim 19, wherein step (h) detects said image candidate when distancefrom the image candidate to said input image is nearest within saidimage space.
 22. The method of claim 20, wherein step (h) uses the leastsquares algorithm to detect said image candidate.
 23. A pose estimationapparatus comprising: a memory for storing a plurality of posecandidates; a three-dimensional model creating mechanism for formulating3D model data indicating a three-dimensional shape of an object andreflectivity or color at every point of said object; an image spacecreating mechanism for successively retrieving a pose candidate fromsaid memory and creating an image space representing brightness valuesof a plurality of two-dimensional images of said object placed in thesame position and orientation as the retrieved pose candidate, whereinsaid brightness values would be obtained if said image object isilluminated under varying lighting conditions; an image candidatedetecting mechanism for detecting an image candidate within said imagespace by using said 3D model data and determining a distance from theimage candidate to an input image; and a selecting mechanism forselecting one of the pose candidates which corresponds to the imagecandidate whose distance to said input image is smallest.
 24. Theapparatus of claim 23, wherein the three-dimensional model creatingmechanism performs the functions of measuring reflectivity or color atevery point of said object under varying illuminations to produce aplurality of illumination variation textures each representing abrightness value of said every point of said object, calculating basistextures approximating said plurality of illumination variationtextures, and formulating said 3D model data so that the image dataindicates said basis textures as well as said three-dimensional shape.25. The apparatus of claim 23, wherein the image candidate detectingmechanism detects said image candidate when distance from the imagecandidate to said input image is nearest within said image space. 26.The apparatus of claim 25, wherein the image candidate detectingmechanism uses the least squares algorithm to detect said imagecandidate.
 27. The apparatus of claim 23, further comprising a posevariant creating mechanism for creating a plurality of pose variantsfrom each of said pose candidates such that each of the pose variants isdisplaced in position and orientation by a predetermined amount fromsaid each pose candidate, and wherein said image space creatingmechanism creates said image space for each of said pose variants. 28.The apparatus of claim 27, wherein each of said pose variants is axiallyand rotationally displaced by said predetermined amount in athree-dimensional coordinate system over a specified range centeredabout said each pose candidate.
 29. The apparatus of claim 28, whereinsaid specified range is determined based on a possible error of saideach pose candidate.
 30. A pose estimation apparatus comprising: a firstmemory for storing a plurality of pose candidates; a 3D model dataformulating mechanism for formulating 3D model data for indicating athree-dimensional shape of an object and reflectivity or color at everypoint of said object; a pose variants creating mechanism forsuccessively reading a pose candidate from said first memory andcreating a plurality of pose variants of the retrieved pose candidatesuch that each of the pose variants is displaced in position andorientation by a predetermined amount from the retrieved pose candidate,and storing the pose variants in a second memory; an image spacecreating mechanism for successively reading a pose variant from saidsecond memory and creating an image space representing brightness valuesof a plurality of two-dimensional images of said object placed in thesame position and orientation as the retrieved pose variant, whereinsaid brightness values would be obtained if said object is illuminatedunder varying lighting conditions; an image candidate detecting andselecting mechanism for successively detecting, in correspondence to theretrieved pose variant, an image candidate within said image space byusing said 3D model data, determining a distance from the imagecandidate to an input image, and selecting one of the pose candidatescorresponding to the image candidate whose distance to said input imageis smallest; and a pose candidate comparing and replacing mechanism forcomparing the selected pose candidate with a previously selected posecandidate, and replacing the previously selected pose candidate with thecurrently selected pose candidate if the currently selected posecandidate is better than the previously selected pose candidate untilthe previously selected pose candidate is better than the currentlyselected pose candidate.
 31. The apparatus of claim 30, wherein thethree-dimensional model creating mechanism performs the functions of:measuring reflectivity or color at every point of said object undervarying illuminations to produce a plurality of illumination variationtextures each representing a brightness value of said every point ofsaid object; calculating basis textures approximating said plurality ofillumination variation textures; and formulating said 3D model data sothat the image data indicates said basis textures as well as saidthree-dimensional shape.
 32. The apparatus of claim 30, wherein theimage candidate detecting mechanism detects said image candidate whendistance from the image candidate to said input image is nearest withinsaid image space.
 33. The apparatus of claim 32, wherein the imagecandidate detecting mechanism uses the least squares algorithm to detectsaid image candidate.
 34. A pose estimation apparatus comprising: athree-dimensional model creating mechanism for formulating 3D model dataindicating a three-dimensional shape of an object and reflectivity orcolor at every point of said object; a feature extracting mechanism forextracting feature points from said object and extracting feature pointsfrom an input image; a pose candidate creating mechanism for creating aplurality of pose candidates from the extracted feature points of saidobject and the extracted feature points of said input image and storingthe pose candidates in a memory; a image space creating mechanism forcreating, for each of said pose candidates, an image space representingbrightness values of a plurality of two-dimensional images of saidobject placed in the same position and orientation as said each posecandidate, wherein said brightness values would be obtained if saidobject is illuminated under varying lighting conditions; an imagecandidate detecting mechanism for detecting, for each of said posecandidates, an image candidate within said image space by using said 3Dmodel data and determining a distance from the image candidate to saidinput image; and an image candidate selecting mechanism for selectingone of the pose candidates which corresponds to the image candidatewhose distance to said input image is smallest.
 35. The apparatus ofclaim 34, wherein the three-dimensional model creating mechanismperforms measuring reflectivity or color at every point of said objectunder varying illuminations to produce a plurality of illuminationvariation textures each representing a brightness value of said everypoint of said object, calculating basis textures approximating saidplurality of illumination variation textures, and formulating said 3Dmodel data so that the image data indicates said basis textures as wellas said three-dimensional shape.
 36. The apparatus of claim 34, whereinthe image candidate detecting mechanism detects said image candidatewhen distance from the image candidate to said input image is nearestwithin said image space.
 37. The apparatus of claim 36, wherein theimage candidate detecting mechanism uses the least squares algorithm todetect said image candidate.
 38. The apparatus of claim 34, furthercomprising a pose variant creating mechanism for creating a plurality ofpose variants from each of said pose candidates such that each of thepose variants is displaced in position and orientation by apredetermined amount from said each pose candidate, and wherein saidimage space creating mechanism creates said image space for each of saidpose variants.
 39. The apparatus of claim 38, wherein each of said posevariants is axially and rotationally displaced by said predeterminedamount in a three-dimensional coordinate system over a specified rangecentered about said each pose candidate.
 40. The apparatus of claim 39,wherein said specified range is determined based on a possible error ofsaid each pose candidate.
 41. A pose estimation apparatus comprising: amemory; a 3D model data formulating mechanism for formulating 3D modeldata indicating a three-dimensional shape of an object and reflectivityor color at every point of said object; a feature points extracting andestimating mechanism for extracting feature points from said object andextracting feature points from an input image and estimating a possibleerror of the extracted feature points of said input image; a posecandidate creating mechanism for creating a plurality of pose candidatesfrom the extracted feature points of said object and the extractedfeature points of said input image and storing the pose candidates insaid memory; a pose variants creating mechanism for successivelyretrieving a pose candidate from said memory and creating a plurality ofpose variants from the retrieved pose candidate such that each of thepose variants is displaced in position and orientation by apredetermined amount from the retrieved pose candidate over a rangedetermined by the estimated possible error; an image space creatingmechanism for creating, for each of said pose variants, an image spacerepresenting brightness values of a plurality of two-dimensional imagesof said object placed in the same position and orientation as said eachpose variant, wherein said brightness values would be obtained if saidimage object is illuminated under varying lighting conditions; and animage candidate detecting and selecting mechanism for detecting, foreach said pose variant, an image candidate within said image space byusing said 3D model data, determining a distance from the imagecandidate to said input image and selecting one of the pose candidatescorresponding to the image candidate whose distance to said input imageis smallest.
 42. The apparatus of claim 41, wherein the 3D model dataformulating mechanism performs the functions of: measuring reflectivityor color at every point of said object under varying illuminations toproduce a plurality of illumination variation textures each representinga brightness value of said every point of said object; calculating basistextures approximating said plurality of illumination variationtextures; and formulating said 3D model data so that the image dataindicates said basis textures as well as said three-dimensional shape.43. The apparatus of claim 41, wherein said image candidate detectingand selecting mechanism detects said image candidate when distance fromthe image candidate to said input image is nearest within said imagespace.
 44. The apparatus of claim 43, wherein said image candidatedetecting and selecting mechanism uses the least squares algorithm todetect said image candidate.
 45. A computer-readable storage mediumcontaining a computer-executable program which comprises the steps of:a) formulating 3D model data indicating a three-dimensional shape of anobject and reflectivity or color at every point of said object; b)creating, for each of said pose candidates, a image space representingbrightness values of a two-dimensional image object of same position andorientation as said each pose candidate, which brightness values wouldbe obtained if said image object is illuminated under varying lightingconditions; c) detecting, for each of said pose candidates, an imagecandidate within said image space by using said 3D model data anddetermining a distance from said image candidate to an input image; andd) selecting one of the pose candidates which corresponds to the imagecandidate whose distance to said input image is smallest.
 46. Thecomputer-readable storage medium of claim 45, wherein step (a) comprisesthe steps of measuring reflectivity or color at every point of saidobject under varying illuminations to produce a plurality ofillumination variation textures each representing a brightness value ofsaid every point of said object, calculating basis texturesapproximating said plurality of illumination variation textures, andformulating said 3D model data so that the image data indicates saidbasis textures as well as said three-dimensional shape.
 47. Thecomputer-readable storage medium of claim 45, wherein step (c) detectssaid image candidate when distance from the image candidate to saidinput image is nearest within said image space.
 48. Thecomputer-readable storage medium of claim 47, wherein step (c) uses theleast squares algorithm to detect said image candidate.
 49. Thecomputer-readable storage medium of claim 45, further comprising thestep of creating a plurality of pose variants from each of said posecandidates such that each of the pose variants is displaced in positionand orientation by a predetermined amount from said each pose candidate,and wherein step (b) creates said image space for each of said posevariants.
 50. The computer-readable storage medium of claim 49, whereineach of said pose variants is axially and rotationally displaced by saidpredetermined amount in a three-dimensional coordinate system over aspecified range centered about said each pose candidate.
 51. Thecomputer-readable storage medium of claim 50, wherein said specifiedrange is determined based on a possible error of said each posecandidate.
 52. A computer-readable storage medium containing acomputer-executable program which comprises the steps of: a) formulating3D model data indicating a three-dimensional shape of an object andreflectivity or color at every point of said object; b) successivelyreading one of said pose candidates from said memory; c) creating aplurality of pose variants from said one pose candidate such that eachof the pose variants is displaced in position and orientation by apredetermined amount from said one pose candidate; d) creating, for eachof said pose variants, an image space representing brightness values ofa plurality of two-dimensional images of said object placed in the sameposition and orientation as said each pose variant, wherein saidbrightness values would be obtained if said object is illuminated undervarying lighting conditions; e) detecting, for each said pose variant,an image candidate within said image space by using said 3D model dataand determining a distance from the image candidate to an input image;f) repeating steps (b) to (e) to produce a plurality of said imagecandidates for each of said pose candidates; g) selecting one of thepose candidates corresponding to the image candidate whose distance tosaid input image is smallest; h) comparing the pose candidate selectedby step (g) with a previously selected pose candidate; and i) replacingthe previously selected pose candidate with the pose candidate currentlyselected by step (g) if the currently selected pose candidate is betterthan the previously selected pose candidate, and repeating steps (b) to(g) until the previously selected pose candidate is better than thecurrently selected pose candidate.
 53. The computer-readable storagemedium of claim 52, wherein step (a) comprises the steps of measuringreflectivity or color at every point of said object under varyingilluminations to produce a plurality of illumination variation textureseach representing a brightness value of said every point of said object,calculating basis textures approximating said plurality of illuminationvariation textures, and formulating said 3D model data so that the imagedata indicates said basis textures as well as said three-dimensionalshape.
 54. The computer-readable storage medium of claim 52, whereinstep (e) detects said image candidate when distance from the imagecandidate to said input image is nearest within said image space. 55.The computer-readable storage medium of claim 54, wherein step (e) usesthe least squares algorithm to detect said image candidate.
 56. Acomputer-readable storage medium containing a computer-executableprogram which comprises the steps of: a) formulating 3D model dataindicating a three-dimensional shape of an object and reflectivity orcolor at every point of said object; b) extracting feature points fromsaid object and extracting feature points from an input image; c)creating a plurality of pose candidates from the extracted featurepoints of said object and the extracted feature points of said inputimage and storing the pose candidates in a memory; d) creating, for eachof said pose candidates, an image space representing brightness valuesof a plurality of two-dimensional images of said object placed in thesame position and orientation as said each pose candidate, wherein saidbrightness values would be obtained if said image object is illuminatedunder varying lighting conditions; e) detecting, for each of said posecandidates, an image candidate within said image space by using said 3Dmodel data and determining a distance from the image candidate to saidinput image; and f) selecting one of the pose candidates whichcorresponds to the image candidate whose distance to said input image issmallest.
 57. The computer-readable storage medium of claim 56, whereinstep (a) comprises the steps of measuring reflectivity or color at everypoint of said object under varying illuminations to produce a pluralityof illumination variation textures each representing a brightness valueof said every point of said object, calculating basis texturesapproximating said plurality of illumination variation textures, andformulating said 3D model data so that the image data indicates saidbasis textures as well as said three-dimensional shape.
 58. Thecomputer-readable storage medium of claim 56, wherein step (e) detectssaid image candidate when distance from the image candidate to saidinput image is nearest within said image space.
 59. Thecomputer-readable storage medium of claim 58, wherein step (e) uses theleast squares algorithm to detect said image candidate.
 60. Thecomputer-readable storage medium of claim 57, further comprising thestep of creating a plurality of pose variants from each of said posecandidates such that each of the pose variants is displaced in positionand orientation by a predetermined amount from said each pose candidate,and wherein step (c) creates said image space for each of said posevariants.
 61. The computer-readable storage medium of claim 60, whereineach of said pose variants is axially and rotationally displaced by saidpredetermined amount in a three-dimensional coordinate system over aspecified range centered about said each pose candidate.
 62. Thecomputer-readable storage medium of claim 61, wherein said specifiedrange is determined based on a possible error of said each posecandidate.
 63. A computer-readable storage medium containing acomputer-executable program which comprises the steps of: a) formulating3D model data indicating a three-dimensional shape of an object andreflectivity or color at every point of said object; b) extractingfeature points from said object and extracting feature points from aninput image; c) estimating a possible error of the extracted featurepoints of said input image; d) creating a plurality of pose candidatesfrom the extracted feature points of said object and the extractedfeature points of said input image and storing the pose candidates in amemory; e) successively reading one of said pose candidates from saidmemory; f) creating a plurality of pose variants from said one posecandidate such that each of the pose variants is displaced in positionand orientation by a predetermined amount from said one pose candidateover a range determined by said possible error estimated by step (c); g)creating, for each of said pose variants, an image space representingbrightness values of a plurality of two-dimensional images of saidobject placed in the same position and orientation as said each posevariant, wherein said brightness values would be obtained if said imageobject is illuminated under varying lighting conditions; h) detecting,for each said pose variant, an image candidate within said image spaceby using said 3D model data and determining a distance from the imagecandidate to said input image; i) repeating steps (e) to (h) to producea plurality of said image candidates for each of said pose candidates;and j) selecting one of the pose candidates corresponding to the imagecandidate whose distance to said input image is smallest.
 64. Thecomputer-readable storage medium of claim 63, wherein step (a) comprisesthe steps of measuring reflectivity or color at every point of saidobject under varying illuminations to produce a plurality ofillumination variation textures each representing a brightness value ofsaid every point of said object calculating basis textures approximatingsaid plurality of illumination variation textures, and formulating said3D model data so that the image data indicates said basis textures aswell as said three-dimensional shape.
 65. The computer-readable storagemedium of claim 63, wherein step (h) detects said image candidate whendistance from the image candidate to said input image is nearest withinsaid image space.
 66. The computer-readable storage medium of claim 65,wherein step (h) uses the least squares algorithm to detect said imagecandidate.